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Background / Context: 

A recent report by the Mathematics Advisory Panel referred to algebra as a “gateway” to later 
achievement (National Mathematics Advisory Panel, 2008). To address the problem of low 
academic performance in algebra, an increasing number of states and districts have started to 
implement a policy of requiring algebra for all students in ninth-grade or earlier. The rationale is 
that providing all students with an opportunity to learn algebra will likely improve math 
achievement across the board. However, the well-intended curricular policy may have 
unintended consequences for some students if the change of curricular offerings leads to a 
reorganization of all math classrooms in a school. 

This policy can be viewed as an effort to eliminate tracking in mathematics instruction. 
Conventionally, secondary school students are placed in different sequences of course-taking 
based on their initial academic skills, interests, and future occupational orientations (Powell, 
Farrar, & Cohen, 1985). The tracking system has been widely criticized for increasing 
educational inequality especially because low-track classrooms are characteristic of low-level 
content, low expectations, and often low engagement (Gamoran & Mare, 1989; Lucas, 1999; 
Oakes, 1985, 2005; Page, 1991). For example, under the tracking system, lower ability 9* 
graders tend to take remedial math together with students at a similar ability level while their 
higher ability peers are taking algebra. To eliminate tracking would require enrolling students 
from mixed ability levels in the same algebra classes. 

An earlier study has shown that the algebra-for-all policy adopted by the Chicago Public Schools 
in 1997 indeed increased algebra enrollment of many lower ability 9th graders who would 
otherwise have taken remedial math. Researchers did not find a policy effect on lower ability 
students’ math achievement. However, lower ability students’ grades and passing rates seemed to 
have declined after the policy was introduced (Allensworth, Nomi, Montgomery, and Lee, 2010). 
In the meantime, although the policy did not change the algebra enrollment of higher ability 
students, it appeared to have an unintended negative impact on the math achievement of those 
who would have taken algebra regardless of the policy (Nomi, 2010). 

We reason that the negative effect on higher ability students’ achievement and that on lower 
ability students’ grades and passing rates are at least partly attributable to policy-induced 
changes in peer ability composition in 9* grade algebra classes. When schools increased algebra 
enrollment and eliminated remedial math classes, lower ability students who would have taken 
remedial math were placed in algebra classes together with higher ability students who would 
have taken algebra regardless of the policy. In general, the heterogeneity of student ability 
increased while the average ability decreased in algebra classes as a result of the policy. 

Average peer ability represents the amount of math knowledge and skills collectively brought by 
students in a class. In theory, even when the curriculum is given, ability composition of a class 
may nonetheless influence instructional content, pace, participation structure, peer interactions, 
and evaluation, which may subsequently influence a student’s math learning and relative 
standing in class. Hence changes in algebra class composition may mediate the effect of the 
algebra-for-all policy on students’ math outcomes. Moreover, the mediation effect may depend 
on the direction and magnitude of the change in peer ability for each individual student given his 
or her incoming skills. 
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Specifically, we hypothesize that experiencing a decline in average peer ability may negatively 
affect the math learning of higher ability students. This is because teachers tend to gear 
instruction to the middle of the ability distribution in a heterogeneous class by reducing content 
coverage or slowing down the pace in de-tracked algebra classrooms (Rosenbaum, 1999). If the 
policy could be implemented without changing algebra class composition, presumably the policy 
should have little impact on higher ability students’ math learning. 

For lower ability students, we hypothesize that experiencing a rise in average peer ability may 
have mixed effects as it may affect their social-emotional process and learning opportunities in 
opposite ways. Being placed in the same class with higher ability peers may heighten peer 
competition, increase anxiety for failure, and lower one’s self-esteem due to unfavorable social 
comparisons. This mechanism is related to the so-called “institutional effect” and “social effect” 
of instructional organization in the sociology literature (Gamoran, 1986; Pallas, Entwisle, 
Alexander, and Stluka, 1994). Yet gaining exposure to higher level content in algebra classes is 
expected to advance lower ability students’ math learning unless the changed course content is 
beyond their reach. 

Purpose / Objective / Research Question / Focus of Study: 

To explain the negative effects of the policy that required algebra for all 9* grade students in 
Chicago Public Schools, our causal questions focus on the mediating role of changes in peer 
ability composition induced by the policy (See Figure 1 for the causal diagram). Specifically, we 
address the following research questions: (1) Did change in peer ability mediate the policy 
effects on student outcomes? (2) What would be the direct effects of the policy on student 
outcomes if schools did not change class composition? (3) Did the mediated effects (i.e., indirect 
effects) and the direct effects of the policy depend on the direction of change in peer ability for 
individual students? 

Setting: 

Chicago has the third-largest school system in the United States. The student population is about 
50% African-American, 38% Latino, 9% White, and 3% Asian. Approximately 85% of students 
are eligible for free/reduced priced lunches. Nearly 55% of students scored below the national 
median on ITBS and a quarter of student failed in mathematics prior to the 1997 policy. 

Population / Participants / Subjects: 

Our analyses include all the CPS high schools in existence before and after the policy was 
implemented (n=59 schools). The study population consists of six cohorts of first-time 9th- 
graders enrolled in CPS high schools between 1994 and 1999 with approximately 20,000 
students in each cohort. 

Intervention / Program / Practice: 

The intervention under study is the curricular policy mandating college-preparatory coursework 
for all high school students. Prior to 1997, students were required to complete two to three years 
of mathematics in any subject and many students began high school with remedial coursework 
(e.g., pre-algebra or general mathematics). In 1997 the Chicago Public Schools (CPS) raised 
graduation standards requiring Algebra I in the ninth grade, followed by Geometry and Algebra 
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II in the subsequent years. We restrict our attention to the requirement of algebra enrollment for 
all 9* graders. 

Research Design: 

This study uses interrupted time series design to identify the causal effects of the algebra-for-all 
policy that presented an exogenous shock to the entire school system. The research design can be 
viewed as a natural experiment as the 1997 policy left little room for self selection of course 
enrollment. The treatment group includes the three cohorts of students who were enrolled in the 
9* grade after 1997; and the control group includes the three cohorts of students enrolled before 
1997. Additionally, to minimize threats to internal validity such as concurrent historical events or 
change in instrumentation, we use a third comparison group — students who attended schools that 
had already offered algebra for all students prior to 1997. Assuming that these schools were not 
affected by the system-wide algebra-for-all policy, we intend to remove the confounding of the 
time-varying factors by comparing the pre-post differences in outcomes between students in 
schools unaffected by the policy and those in schools that increased algebra enrollment. 

Data Collection and Analysis: 

Notation. We use T to denote the system-wide policy. Let t=l denote the algebra-for-all policy 
and t = 0 for the control condition. The mediator of focal interest to us is the ability composition 
of a math class. We use C{t) to represent average peer ability as a function of the algebra-for-all 
policy. In assessing the mediated policy effect on math achievement, we compute peer ability as 
the class average math pretest score relative to the school average score in a given year. This 
measure of class average ability is independent of between-school differences in student 
academic composition and of within-school year-to-year variation in student academic 
composition. In assessing the mediated policy effect on math grades and course failure, we 
compute peer ability as the class average math pretest score relative to a focal student’s pretest 
score. Because the policy was directly targeted at changing math course-taking of lower ability 
9* graders, we use M{t) to represent math course-taking as a function of the policy. Let M{t) = 1 
if a student takes algebra and 0 otherwise for t = 0, 1. Under the algebra-for-all policy, we have 
that M(l) = 1 for all students. The math outcomes of interest include standardized math test 
score, math grade, and whether a student passes or fails in a math class. A student’s potential 
math outcomes are represented by Y{t) which is equivalent to Y{t, M(t), C(t)). We use X to denote 
a vector of student and school characteristic measures. 

Data sources. This study draws from multiple data sources provided by the Chicago Public 
Schools. Course transcripts and semester grade files contain information on students’ course 
enrollment. Administrative records contain student demographic information including gender, 
age, race/ethnicity, special-education status, and residential mobility. Student socio-economic 
variables were constructed using the 2000 U.S. census block-level data linked to students’ home 
addresses. We use student scores on the Iowa Tests of Basic Skills from third to eighth grade to 
measure a student’s incoming ability. High school achievement test scores come from the Tests 
of Academic Proficiency (TAP) given at the end of the 9* grade (see Table 1 for details). 
School-level information includes school type, size, demographics, and academic composition. 

Total effect. The total effect of the algebra-for-all policy is defined as the average difference 
between the potential outcome associated with the algebra-for-all policy and the potential 
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outcome that is not associated with this policy. This is represented by E\Y{t=\) - T(t=0)] = 

E\Y{\, M(l), C(l)) - T(0, M(0), C(0))]. 

Natural direct effect and natural indirect effect. Recent statistics literature has decomposed the 
total effect into a natural direct effect and a natural indirect effect (Pearl, 2001; Robins & 
Greenland, 1992). In this study, the algebra- for- all policy was expected to change course offering 
and course taking and thereby change students’ math outcomes even without changing peer 
ability. The natural direct effect of interest here is the policy effect on student outcomes not to be 
attributed to policy-induced changes in peer ability. A student’s counterfactual math outcome 
under the policy with peer ability counterfactually remained unchanged by the policy is 
represented by 7(1, M(l), C(0)). Hence, the natural direct effect of the policy with peer ability 
remaining counterfactually unchanged is represented by £'[7(1, M(l), C(0)) - 7(0, M(0), C(0))]. 
The natural indirect effect is the change in student outcomes that is attributable to the change in 
peer ability induced by the policy. In other words, it is the policy effect mediated by the change 
in peer ability and is represented by £[7(1, M(l), C(l)) - 7(1, M(l), C(0))]. 

Student subpopulations. As noted earlier, we hypothesize that policy-induced changes in peer 
ability may affect higher ability and lower ability students differently. Hence we define direct 
and indirect effects of the policy for subpopulations of students that are relatively homogeneous 
in responding to the change in policy. Subpopulation memberships are jointly determined by a 
student’s expected change in average peer ability £[C(1) - C(0) I X] and the expected change in 
course-taking £[M(1) - M(0) I X] given the student’s incoming abilities in reading and math, 
student demographic characteristics, and school membership. For example, we expect that higher 
ability students would have a negative value in peer ability change and would have no change in 
course-taking and that lower ability students would have a positive value in peer ability change 
as well as a positive value in course-taking change. 

Analytic strategy. To estimate the direct and indirect effects moderated by student prior ability, 
we use the ratio-of-mediator-probability weighting method (Hong, 2010). Conventional methods 
such as path analysis or structural equation modeling require the assumption of no interaction 
between treatment and mediator (Holland, 1988). These methods are not applicable in this study 
because we are particularly interested in examining the interaction between the algebra-for-all 
policy and the change in peer ability as a mediator. Recent statistical advances relax this 
assumption typically within the linear or nonlinear regression framework (Pearl, 2010; Petersen, 
Sinisi, & van der Laan, 2006; VanderWeele, 2009) with few exceptions (Imai, Keele, & 
Yamamoto, 2010). Among other limitations, none of these strategies provide simultaneous 
estimation of the confidence intervals for the point estimates of natural direct and indirect 
effects. Our non-parametric weighting approach not only relaxes the assumption of no treatment- 
by-mediator interaction but also enhances the robustness of estimation results when there are 
possible model misspecifications. In essence, we estimate the marginal mean of each 
counterfactual outcome for each subpopulation of students by weighting every student enrolled 
in high school after 1997 such that the weighted distribution of their peer ability under the policy 
approximates their counterfactual distribution prior to 1997. The weight to be applied for this 
purpose is a ratio of the conditional probability of pre-policy peer ability to the conditional 
probability of post-policy peer ability. Most importantly, the weighting strategy allows us to 
estimate the average counterfactual outcome £[7(1, M(l), C(0))]. Recent results have shown that 
weighting on the basis of propensity score stratification produces causal effect estimates that are 
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robust to misspecifications of the propensity score model (Hong, in press). After dividing the 
sample into a number of strata on the propensity score for pre-policy peer ability and again 
stratifying the sample on the propensity score for post-policy peer ability, we estimate the ratio- 
of- mediator-probability weight for each student enrolled in high school after 1997. The weighted 
outcome model is a function of the subpopulation-specific direct and indirect effects with 
minimal model-based assumptions. We use estimated robust standard errors to compute 
respective confidence intervals. 

Findings / Results: 

The analysis of pre-policy course-taking has shown that almost all 9th graders whose incoming 
math skills were higher than 0.5 standard deviations above the overall average took algebra in 
the absence of the policy. Hence the policy would not change these students’ algebra enrollment. 
However, the results indicated that these same students would have experienced a decline in peer 
ability on average had they instead entered high school after 1997. Among students whose 
incoming math skills were no more than 0.5 standard deviations above the overall average, those 
who were more likely to take remedial math prior to the policy showed a higher likelihood of 
experiencing an increase in peer ability along with a change in course taking had they instead 
entered high school after 1997. We are currently conducting mediation analysis. 

Conclusions: 

Student achievement in algebra is one of the key predictors for academic success. Algebra 
typically has the highest failure rate among all 9*-grade courses and is thus directly tied to high 
school graduation rate (Allensworth and Easton, 2007). This study shows that a well-intended 
policy of universalizing algebra education through changing not only the curriculum but also 
instructional organization can have mixed effects on these important outcomes. We apply a new 
analytic strategy to population data that enables us to answer the questions about causal 
mechanisms that cannot be answered by standard regression-based methods such as path analysis 
or structural equation modeling. By showing how changes in peer ability mediate the policy 
effects on student outcomes, this study contributes to the extant research on tracking and de- 
tracking. While tracking has been widely criticized for increasing educational inequality and de- 
tracking often seen as a preferable alternative (Boaler & Staples, 2008; Oakes, 2005; Rubin, 
2008), some researchers have been concerned about the practical difficulties with de-tracking 
(Rosenbaum, 1999). Results from this study empirically inform our understanding of how 
students are influenced differently by changes in peer ability composition as a result of de- 
tracking. In particular, simply eliminating tracking would have a negative impact on higher 
ability students as a decline in peer ability composition might lead to a reduction in academic 
demands within a class. For lower ability students whose course taking were to be changed by 
the policy, attending algebra classes together with higher ability students tended to increase their 
failure rate as an increase in peer ability would be accompanied by a decrease in a lower ability 
student’s relative standing in class. Moreover, due to these students’ relatively weak math 
background, learning challenging math content without additional support might also contribute 
to their failure in algebra classes (also see, Nomi & Allensworth, 2009), which could be an 
explanation for the direct effects of the policy on lower ability students’ grades and failure rate 
unaccounted for by the change in peer ability. Revealing these subtleties in the causal 
mechanisms brings us important new insights in evaluating systemic policy initiatives. 



2011 SREE Conference Abstract Template 



5 




Appendices 



Appendix A. References 

Allensworth, E., & Easton, J. Q. (2007). What matters for staying on-track and graduating in 
Chicago Public Schools. Chicago, IE: the Consortium on Chicago School Research. 

Allensworth, E., Nomi, T., Montgomery, N., & Eee, V. E. (2010). College preparatory 

curriculum for all: Academic consequences of requiring Algebra and English I for ninth 
graders in Chicago. Education Evaluation and Policy Analysis, 31{A), 367-391. 

Boaler, J., & Staples, M. (2008). Creating mathematical futures through an equitable teaching 
approach: The case of Railside School. Teachers College Record 110, 608-645. 

Gamoran, A. (1986). Instructional and Institutional Effects of Ability Grouping. Sociology of 
Education, 59, 185-198. 

Gamoran, A., & Mare, R. D. (1989). Secondary school trackign and educational inequality: 

Compensation, reinforcement, or neutrality? The American Journal of Sociology, 94(5), 
1146-1183. 

Holland, P. W. (1988). Causal inference, path analysis, and recursive structural equation models 
(with discussion). In C. C. Clogg (Ed.), Sociological Methodology (pp.449-493). 
Washington, DC: American Sociological Association. 

Hong, G. (2010). Ratio of mediator probability weighting for estimating natural direct and 

indirect effects. In 2010 Proceedings of the American Statistical Association, Biometrics 
Section, Alexandria, VA: American Statistical Association. 

Hong, G. (In press). Marginal mean weighting through stratification: Adjustment for selection 
bias in multi-level data. Journal of Educational and Behavioral Statistics. 

Imai, K., Keele, E., & Yamamoto, T. (2010). Identification, inference and sensitivity analysis for 
causal mediation effects. Statistical Science, 25(1), 51-71. 

Eucas, S. R. (1999). Tracking Inequality: Stratification and Mobility in American High School. 
New York: Teachers College Press. 

National Mathematics Advisory Panel. (2008). The Einal report of the National Mathematics 
Advisory Panel. Washington DC: U.S. Department of Education. 

Nomi, T. (2010). The Unintended Consequences of an Algebra-for-All policy on High-skill 

Students: the Effects on Instructional Organization and Students’ Academic Outcomes. 
Paper presented at the Society for Research on Educational Effectiveness. Washington, 
DC. 

Nomi, T., & Allensworth, E. (2009). Double-Dose" Algebra as an Alternative Strategy to 
Remediation: Effects on Students' Academic Outcomes. Journal of Research on 
Educational Effectiveness, 2(2), 111-148. 

Oakes, J. (1985). Keeping track: How schools structure inequality. Birmingham, N.Y.: Vail- 
Ballou Press. 

Oakes, J. (2005). Keeping track: How schools structure inequality (2nd ed. ed.): New Haven: 
Yale University Press. 

Page, R. N. (1991). Lower Track Classroom: A Curricular and Cultural Perspective. New York: 
Teachers College Press. 

Pallas, A. M., Entwisle, D. R., Alexander, K. E., & Stluka, E. M. (1994). Ability-Group Effects: 
Instructional, Social, or Institutional? Sociology of Education, 67, 27-46. 



2011 SREE Conference Abstract Template 



A-1 




Pearl, J. (2001). Direct and indirect effects. In Proceedings of the American Statistical 

Association Joint Statistical Meetings. Minn, MN: MIRA Digital Publishing, 1572-1581, 
August 2005. 

Pearl, J. (2010). The mediation formula: A guide to the assessment of causal pathways in non- 
linear models. Los Angeles, CA: University of California, Los Angeles. Technical report 
R-363, July 2010. 

Peterson, M. L., Sinisi, S. E., & van der Laan, M. J. (2006). Estimation of direct causal effects. 
Epidemiology, 17(3), 276-284. 

Powell, A. G., Farrar, E., & Cohen, D. K. (1985). The shopping mall high school: Winners and 
losers in the educational marketplace. . Boston: Houghton Mifflin. 

Robins, J. M. & Greenland, S. (1992). Identifiability and exchangeability for direct and indirect 
effects. Epidemiology, 3(2), 143-155. 

Rosenbaum, J.E. (1999). If tracking is bad, is detracking better? American Educator, 23(4), 24- 
29, 47. 

Rubin, B. C. (2008). Detracking in context: How local constructions of ability complicate equity. 
Teachers College Record, 110(3), 646-699. 

VanderWeele, T. (2009). Marginal structural models for the estimation of direct and indirect 
effects. Epidemiology, 20, 18-26. 



201 1 SREE Conference Abstract Template 



A-2 




Appendix B. Tables and Figures 



Figure 1. 

Causal Relationships among the Algebra-for-All Poliey, Peer Ability, and Math Outeomes 




201 1 SREE Conference Abstract Template 



B-1 







Table 1. 

Variable descriptions 



Students variables 

9‘^ grade outcomes 
GPA 

Course failure 
Test scores 

Other covariates 
Student ability 

Algebra enrollment 
Special education 
Gender 

Race/Ethnicity 

SES 



Mobility 
Age at HS entry 



Classroom covariates 
Peer ability level 
Cohort covariates 

Cohort 

Cohort average 
ability 



Average math GPA on a 4 point scale 

A dummy variable indicating whether students failed math courses 
Math test scores measured in 9* grade spring semester (TAP) 



Eatent ability scores in math and English were created for each student 
using his/her ITBS history from 3’^‘^ grade through 8* grade 
(standardized across all cohorts with a mean of 0 and SD of 1) 

A dummy variable with l=enrolling in algebra and O=otherwise 
A dummy variable with 1= special education students and O=otherwise 
A dummy variable with l=male and 0=female 
A set of dummy variables indicating African-American (ref. group), 
Asian, Hispanic, and White. 

Two variables constructed from the U.S. census data on students’ 
residential block groups: 1) Concentration of poverty (a composite of 
male unemployment rate and % families under the poverty line. 2) 
Social status (a composite of the median family income and the 
average educational attainment. Both were standardized to have a mean 
of 0 and SD of 1 . 

A set of dummy variables: no moves, one move, two or more moves in 
the 3 yrs before high school 

1) Number of months old for high school 

2) a dummy variable indicating if students are slightly old 

3) a dummy variable indicating if students are young for starting high 
school 

Classroom average latent math scores 

A set of dummy variable distinguishing cohorts with the 1994 (i.e., 
base-year) cohort as a omitted group 
Cohort average latent math scores 
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